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Chapter 1 

Introduction 


The Interactive Electronic Technical Manual (IETM) Cost-Benefit Analysis Tool 
is an Excel spreadsheet-based model that compares the costs and benefits of can¬ 
didate IETM deployment alternatives. This tool applies technology from previous 
LMI cost-benefit models for law enforcement 1 and commercial aviation" to 
IETMs. We invite anyone who may benefit from using the tool to read this man¬ 
ual and investigate its current features. 

The project team has discovered that the training or operational organizations that 
use IETMs have not compiled much of the data needed to perform a complete 
cost-benefit analysis. Although this lack of data limits the applicability of the 
model, it represents an opportunity to demonstrate the types of data, and data re¬ 
positories, required for future analyses. By demonstrating our cost-benefit analy¬ 
sis methodology on a notional set of IETM projects and alternatives, the model 
aids the user in identifying the types and formats of data to gather for future 
analyses. 

The current model has three modes of operation: standard analysis, sensitivity 
analysis, and simulation. The tool also has a utility worksheet that helps the user 
to estimate the cost of converting a paper manual, or a low-level electronic man¬ 
ual (ETM), to a higher-level ETM or IETM. A companion utility also helps the 
user to load a new IETM project into the workbook for analysis. 

The remainder of this manual is organized as follows: 

♦ Chapter 2: Description of the Tool explains the layout of the application. 

♦ Chapter 3: Tutorials includes instructions on loading a set of projects, im¬ 
porting data into the application, each of the three modes of operation, and 
the conversion cost utility worksheet. 

♦ Appendix: Description of Variables explains what each variable in the 
analysis represents. 

This tool must be used with caution and discretion. The primary purpose of this 
tool is to demonstrate the features and types of analysis that an IETM cost-benefit 

1 Cooper, W., J. Dukovich, and J. Bouffard, “The Corrections and Law Enforcement Tech¬ 
nology Assessment System (CLETAS),” Proceedings of IEEE International Engineering Man- 
agement Conference, San Juan, May 1999. 

2 Logistics Management Institute, The Aviation System Analysis Capability Air-Carrier Cost- 
Benefit Model Report NS804S1, Eric M. Gaier, Alexander Edlich, Tara S. Santmire, and Earl R. 
Wingrove, January 1999. 
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analysis tool should have, given sufficient access to development, maintenance, 
and training data for a relatively thorough sample of IETMs. Because this body of 
data does not currently exist, the methods in this tool remain largely rudimentary; 
the assumptions that we used to develop these methods may change as we learn 
more. Therefore, the user should take special heed when the tool—particularly its 
default values—disagrees with the user’s personal experience. 




Chapter 2 

Description of the Tool 


This tool compares one or more alternative programs to a historical, baseline pro¬ 
gram. The user can compare up to four alternatives at once with a baseline that the 
user chooses or creates. Typically, all available program data for the baseline and 
the alternatives reside in a Programs sheet. The user selects the baseline and alter¬ 
natives from a Start sheet and clicks a button that loads them onto tables in the 
Start sheet. After adjusting the data (if desired), the user then runs a standard 
analysis, sensitivity analysis, or simulation. To perform a sensitivity analysis or 
simulation, the user must also specify a high and low variant for a set of desired 
parameters in a special Variants sheet prior to running the analysis. All three 
analysis routines generate reports and graphs that the user can access from the 
Start sheet. The tool has five basic components: the Start sheet, the Variants sheet, 
the Programs sheet, Reports sheets, and output graphs. 


Start Sheet 


This sheet (Figure 2-1) is the “command center.” It allows the user to load data in 
preparation for an analysis, modify the data, and run the analysis. It also provides 
controls that send the user to output reports and graphs. It contains data that the 
user loads from the Programs sheet or enters manually. 


Figure 2-1. Start Sheet 
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Programs Sheet 

This sheet (Figure 2-2) holds the raw, historical data for any available programs. 
Because of the lack of data sources, loading a database of programs into this page 
automatically, as originally intended, is infeasible at present. Nevertheless, this 
sheet is useful as a repository for data that will be used in more than one analysis. 

Figure 2-2. Programs Sheet 
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Each program is arranged as a set of cost reports. Each cost report has its own 
date and takes up one row in the Programs sheet. Each report consists of several 
variables that describe how the IETM has progressed in its life cycle since the 
previous report. For example, if maintenance and revisions have cost $10,000 
since the previous report, the user should find that value in the Period Mainte¬ 
nance/Revisions Cost” column in the current cost report’s row. The appendix de¬ 
scribes each variable in the cost report in detail. 

The tool uses the cost report concept so it can compare how the costs and benefits 
accrue over time between the baseline and alternative programs. For example, 
initial development and procurement costs may make an advanced IETM seem 
prohibitively expensive in comparison with a Class IIETM over the first several 
months of the program. Over subsequent years, however, the advanced IETM 
may provide maintenance and training savings that make it cheaper over the long 
term. By comparing the programs with each cost report as a data point, the tool 
can in some cases provide insight into whether an initially more expensive pro¬ 
gram eventually will reach a “break even” point with regard to a less-expensive 
alternative. 

Because the tool directly compares cost reports from a baseline program to cost 
reports from one to four alternative programs, the user must ensure that the time 
interval between each report is consistent for each program in the analysis. We 
have standardized on quarterly time intervals, and some of the utilities in this tool 
assume quarterly time intervals. The user can choose to use annual intervals, 
monthly intervals, or other intervals if desired. Comparing a program that has 
monthly cost reports with a program that has quarterly cost reports is impermissi¬ 
ble, however. Such a comparison will produce meaningless results. To compare 


2-2 




Description of the Tool 


these programs, the user would have to combine the values in the monthly cost 
reports to create quarterly reports or divide each quarterly report evenly to create 
three monthly reports. 

As long as programs have consistent time intervals for their reports, they can be 
compared directly—even if the reports do not share the same date or if one pro¬ 
gram has more cost reports than the other. The tool converts all dollar values into 
constant-year dollars, so dates do not matter. In the event that one program has 
more reports than another, the tool ignores the additional cost reports in the com¬ 
parison. For example, if one program had six cost reports and another had eight, 
the tool would compare the six cost reports of the first program with the first six 
cost reports of the other. 

Variants Sheet 

This sheet (Figure 2-3) allows the user to insert a high and/or low variant for each 
parameter in the analysis. For example, the tool considers student hourly pay in 
calculating training costs associated with an IETM. These costs may vary de¬ 
pending upon the specific students involved; perhaps an analyst believes that the 
actual costs will fall within ±10 percent of the nominal student pay. The user can 
enter these variants as percentages of the nominal value on the Variants sheet and 
then run a sensitivity analysis to see how these variations would affect the out¬ 
come. 


Figure 2-3. Variants Sheet 

Sensitivity Analysis And Simulation Variants Page 
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Report Sheets 

These sheets (Figure 2-4) display the raw output values that an analysis generates. 
These values feed the output graphs. 
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Figure 2-4. Sample Reports 
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-$289,489.60 
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-$35,443.40 
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Sensitivity Analysis Summary Page: Net Savings At Last Period 
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00 0 oooo 0 


Output Graphs 

The model automatically generates output graphs, including bar charts that com¬ 
pare the costs of a baseline program to its alternatives and a chart that compares 
the net savings of each alternative relative to the baseline (Figure 2-5). The sensi¬ 
tivity analysis routine produces charts that show, for each alternative, the net 
savings relative to the baseline and the total cumulative cost (TCC) of the alterna¬ 
tive. These charts—which are called tornado diagrams—show the sensitivity of 
the output variables (net savings, TCC) to variations in each of the input vari¬ 
ables. They sort the variables according to their effect on the output; the most sen¬ 
sitive variable appears at the top, the least sensitive at the bottom. 
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Figure 2-5. Sample Charts 
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Chapter 3 

Tutorials 


Loading Input Data 

The first step in performing a cost-benefit analysis is to load the data you need. 
You do this through controls that are located on the start sheet, under the “Data 
Selection” heading. Figure 3-1 shows these controls. 

Figure 3-1. Data Loading Controls 

Data Selection 


Baseline 


Select baseline IETM by ID: 
Project Name: Empty 
Subsystem: Empty 
IETM Name: Empty 


Use I'^ri dollars. 

Load ifij 


Alternative I 
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IETM Name: Empty _ 

Alternative HI 

Select revised IETM II by ID: 
Project Name: Erapty 

Subsystem: Empty _ 
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Alternative II ^ 

Select revised IETM III by ID: fcT J§j 

Project Name: Empty _ 

Subsystem: Empty _ 

IETM Name: Empty_ 

Alternative IV 

Select revised IETM IV by ID: |o |p 

Project Name: Empty _ 

Subsystem: Emp ty _ 

IETM Name: Empty _ 


This tutorial compares two different alternatives against a baseline. The baseline 
case will be a Class II IETM for an emergency fill pump. The other two cases will 
be for similar pumps. Alternative I represents a Class IV IETM with a training 
program optimized to capitalize on its advanced features. Alternative II represents 
a transition to a Class IV IETM without changing the training program. In this 
tutorial, you select these alternatives, select a base year for the financial data, load 
the data into their respective tables, and inspect the data. 

1. Select 2 in the pull-down box with the “Select baseline IETM by ID” cap¬ 
tion. You should see the lines underneath the caption update automatically 
to appear as shown in Figure 3-2. 
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Figure 3-2. Selecting the Baseline Project 


Baseline 


o 


Select baseline IETM by ID: l?.III! 

Project Name: T UTORIAL Fill System 

Subsystem: Em. Fill Pump _ 

IETM Name: efpoi-u-S _ 


Here’s what happens: The data for each IETM has a unique ID, much like a data¬ 
base key. When you select “2” in the pull-down menu, the spreadsheet retrieves 
the system name, subsystem name, and IETM name from the first line in the Pro¬ 
grams sheet that had a 2 for its ID. 

Note: The data itself has not yet been loaded into the Start sheet table. We do this 
because loading the data from the Programs sheet into the Start sheet takes time. 
The actual data are loaded all at once, using the “Load” button. 

2. Select 1 in the Alternative I IETM ID pull-down menu, and 3 in the 
Alternative II pull-down menu. 

3. Select a fiscal year for the data in the pull-down menu just above the 
“Load” button; choose 1997. 

By selecting a fiscal year, you tell the application to convert any money values 
from the Programs sheet into base-year dollars. The dollar values in the Programs 
sheet are expressed in then-year dollars. 

4. Load the data by clicking the “Load” button. The project information 
should now appear in bold, rather than italic, indicating that the ID in the 
pull-down menu has not changed since the last load. 

5. Review the baseline and alternative data. The data tables for each case are 
located under the Selection Data controls. Each alternative should have 12 
rows of data, which should be filled in completely. 

6. Save your work. 

If you toggle back and forth between the raw data in the Programs sheet and the 
loaded data on the Start sheet, you will observe that the loaded data only show the 
values that the analysis routines need. Additionally, the money values have been 
converted to 1997 dollars. 

Note that you do not need to use the Programs sheet at all. If you wish, you may 
clear the data tables on the Start sheet and enter a case by hand. You also may 
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load a baseline and a set of alternatives and then modify any values in the data 
tables that you desire. For example, you could load the same case from the Pro¬ 
grams sheet as both the baseline and Alternative I and subsequently modify the 
Alternative I data table to reflect some planned modification. 

Importing Data 

The first tutorial uses sample data that are already in the tool. This tutorial ex¬ 
plains how to import your own data into the tool, using a separate utility work¬ 
book. It is possible to add data directly to the Programs sheet. We do not 
recommend this approach, however, because the tool performs no validity check 
on the data. If you use the utility provided for this purpose, it will automatically 
check the data for you and catch most (but not all) input errors. 

To start the tutorial, open the data importing utility. This utility resides in the 
same directory as the tool itself; it should have a name similar to 
“ImportDataUtil.xls.” In this tutorial, you will import a project for an imaginary 
FA-18 avionics IETM. The data values that you will use have been picked at 
random; they are for illustrative purposes only. 

1. Step 1 of the utility is at the top of the UserSheet worksheet. In this step, 
the user enters data that do not change over time: the program name, the 
subsystem name, the tech manual serial number, the technical manual 
class, and the subsystem category. 

>- In the blue cell labeled “Enter the Program Name Here,” enter 
“FA-18”. 

> In the blue cell labeled “Enter the Subsystem Name Here,” enter 
“Avionics.” 

>► In the blue cell labeled “Enter the TM Serial Number Here,” enter 
“001-78-90.” 

>- In the control box labeled “Enter the TM Class Here,” click on the 
“Class IV” radio button. 

>- In the control box labeled “Enter the Subsystem Category Here,” click 
on the “Information Technology Systems” radio button. The work¬ 
sheet should now look like Figure 3-3. 
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Figure 3-3. Step 1—Data Importing Utility 
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If you have a question about an item that you must enter, place your mouse cursor 
over the small red triangles near that item’s label. Doing so will pop up a com¬ 
ment box that describes the item further (see Figure 3-4). 

Figure 3-4. Built-in Comment Box 
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2. In Step 2, you create a time frame for the project by entering a start date 
and a number of time intervals. The start date corresponds to the date 
when program office personnel first report accrued costs for the technical 
manual. The number of quarters will be used to generate dates of subse¬ 
quent cost reports; the tool assumes quarterly cost reporting. (If you want 
to use another time interval, you will have to delete the automatically gen¬ 
erated dates and insert your own dates in Step 3.) Enter “1/1988” for the 
start date and 4 for the number of quarters (see Figure 3-5). 

Figure 3-5. Step 2—Data Importing Utility 


Step 2: Identify time interval 
Explain This 

E-Mil! 

Enter the Start Date here: "" I Jan-1988| 

Enter the number of quarters'" 

here: _ t 


In most cases, it should be acceptable to roll-up or break down cost reports of dif¬ 
ferent time intervals so that they can be entered as quarterly data. For example, it 
is acceptable to roll up three monthly reports or split the data from one semi¬ 
annual report to generate quarterly data. 

3. Next, you will need to enter the cost data for the manual into the datasheet. 
Click the “Initialize Data Sheet” button on the Usersheet worksheet. You 
will see a text box with a warning like the one in Figure 3-6. Click “Yes” 
on the text box to proceed to the data sheet. 

Figure 3-6. Step 3—Data Importing Utility 
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Data Sheet 
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When the data sheet appears on your screen, you will see that any previous data 
have been wiped out. In response to steps 1 and 2, the data sheet has filled four 
lines on the data sheet with the project’s general information and report dates 
(Figure 3-7). The tool automatically generates this data as a shortcut for the user. 
If you wish, you may manually delete and re-enter any information on the 
Datasheet. (This is generally not recommended; a notable exception would be 
when the user wishes to change the time interval from the quarterly interval as¬ 
sumed by the utility.) 

Figure 3-7. Newly Initialized Data Sheet 
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4. This step will be tedious; please be patient. Fill in the data sheet as shown 
in Figures 3-8 and 3-9. If you complete this step correctly, these data will 
pass the data check of Step 5. If you are uncertain about what a column 
heading means, move the cursor over the red triangle at the comer of the 
heading cell. When you are done, return to the user sheet by clicking the 
button at the top-left comer of the sheet. 


Figure 3-8. Data Sheet Entries, Part 1 
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Figure 3-9. Data Sheet Entries, Part 2 
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Step 4 allows you to return to the data sheet without re-initializing the project 
data. No automated data checking occurs here; the purpose of this step is to give 
you a chance to double-check the data-entry work, if you wish. 

5. Step 5 requires the user to designate the name of the IETM CBA tool 
workbook. If this workbook is in a different directory than the data im¬ 
porting utility’s directory, you must specify the path also. For the purpose 
of this tutorial, copy your IETM CBA tool workbook to the working di¬ 
rectory and rename the copy “IETM_Test.xls.” Then enter that name in 
the blue cell labeled “Enter the IETM CBA Tool file name here” (Figure 
3-10). 

Figure 3-10. Step 5—Data Importing Utility 
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6. Click the “Import” button. If you entered the data correctly and completely 
in step 4, you should see a text box that reads “Data Check OK.” If you 
do, click “OK” to continue. If not, you will see an error message explain¬ 
ing what is wrong with the current data cell. When you acknowledge that 
message, you will find yourself on the data sheet, with the affected cell 
highlighted. Correct the cell, return to the user sheet, and click the “Im¬ 
port” button to try again. 


IETM Test.xls 


3-7 















7. When you have successfully loaded the data into the JETM_Test.xls 
workbook, save your work, close the data importing utility, and open the 
IETM_Test.xls workbook. 

8. Go to the Programs worksheet. You should see the project appended to the 
current list of projects in the sheet. It also should have its own unique 
Entry ED (Figure 3-11). 

Figure 3-11. Programs Sheet 
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9. Click the “Update Data” button. This step allows the tool to recognize the 
new data set. It also leaves you at the Start sheet. To verify that the data 
set has been added, click on any of the pull-down boxes. The new entry ID 
should be in the pull-down list. 

10. To erase the new data set, return to the Programs sheet. Clear the data set 
and click the “Update Data” button again. Check a pull-down box on the 
Start sheet to verify that the new data set is no longer in the pull-down list. 

11. Close the IETM_Test.xls workbook. 

Standard Analysis 

A standard analysis computes costs by category for items such as training and 
maintenance. The analysis routine generates three reports and eight output graphs. 
In this tutorial, you will run an analysis and inspect the output. To follow along 
with this tutorial, you will need to have completed the first tutorial, and the 
spreadsheet must be unmodified since. 
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1. Run the analysis by clicking the “Run” button under the “Standard Analy¬ 
sis” heading in the Start sheet. 

2. Look at the graphs. Click the “Graphs” button under the “Standard Analy¬ 
sis” heading in the Start sheet. The form in Figure 3-12 should appear. 

Figure 3-12. Standard Analysis Chart Form 
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3. Click on the third unit cost chart button—the button labeled “Training 
Costs/Grad.” A graph similar to the one in Figure 3-13 should appear. 

Figure 3-13. Unit Training Cost Chart 
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The chart in Figure 3-13 shows the cost, per graduate, to provide training on the 
equipment that the IETM documents. Recall that Alternative I is a Class IVIETM 
with the training curriculum redesigned to take advantage of the IETM’s ad¬ 
vanced capabilities. As a result, the cost per student dropped as the new curricu¬ 
lum took hold and instructors became more effective at implementing it. The 
second alternative had no such curriculum redesign; the numbers show that train¬ 
ing continues with no significant departure from the baseline. 

Keep the following points in mind: 

♦ These numbers, and the numbers throughout this user’s guide, are strictly 
notional. Actual cases may behave differently. 

♦ The time period is not fixed; it depends on the interval between report 
dates for the data. The time step must be consistent, however, across the 
baseline and the alternatives. In this and other tutorials, each time step is 
one calendar quarter. 

4. Click the “Graphs” button in the upper right comer of the chart sheet. 
Doing so hides the current chart and restores the chart selection form. 
Click button 3 under the Total Costs heading in the form—the button la¬ 
beled “Training Costs.” The chart shown in Figure 3-14 should appear. 

Figure 3-14. Total Training Costs Chart 
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Notice the two distinct spikes in the Alternative I costs that do not appear in the 
unit (cost/graduate) training costs. These spikes occur because an extra class 
graduated in quarters 8 and 12 as a result of condensing the curriculum. Starting 
in quarter 5, Alternative I holds five classes a year, compared with four a year for 
Alternative II and the baseline. Although training expenditures are uniform on a 
day-to-day basis, the utility assumes that costs occur when students graduate. 
Therefore, for the first 3 quarters of the year, the actual unit training costs are 
higher for Alternative I than the chart shows; the 4th quarter makes up the differ¬ 
ence. 

5. Now look at the combined total costs. Click the “Graphs” button, then se¬ 
lect button 4 under the “Total Costs” heading in the chart selection form. 
The chart shown in Figure 3-15 should appear. 

Figure 3-15. Total Combined Costs Chart 
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Alternative I and Alternative II have significant initial costs, presumably associ¬ 
ated with development of the IETM. Alternative I has the highest initial costs be¬ 
cause, in addition to the new IETM, the program also overhauls its training 
system. As time goes on, Alternative I has the lower costs because it enjoys sav¬ 
ings from both reduced maintenance costs and reduced training costs. Although 
Alternative II exhibits only lower maintenance costs—because it is an easier-to- 
use IETM—it still generates some savings. To find out exactly how much savings 
each alternative provides with respect to the baseline, look at the net savings 
chart. 
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6. Click the “Graphs” button, then click button 5 under the “Total Costs” 
heading on the chart selection form. The chart shown in Figure 3-16 
should appear. 

Figure 3-16. Net Savings Against Baseline 


fy: 1997 Net Savings From Baseline 
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Clearly, Alternative I is the best choice because it saves money with respect to the 
baseline. It breaks even during quarter 7, and it saves close to $860,000 by the last 
quarter of available data. Alternative II starts to gain on the baseline by quarter 5, 
but the savings it exhibits in quarters 5 through 12 do not bring it back to the 
break-even point. 

Program Size: An Aside 

The IETM Cost-Benefit Analysis Tool is flexible enough to perform several types 
of analyses. For example, the user can do a notional study by loading a program 
as both a baseline and an alternative case and modifying the alternative data to 
reflect the expected effects of some proposed change. This technique allows the 
user to answer questions such as, “Assuming that we can save 8 percent per unit 
per quarter on maintenance costs with a new IETM, but the IETM will cost an 
extra 4 percent per quarter for revisions and upkeep, does pursuing the alternative 
make sense?” 

To avoid making assumptions about new technologies, the user could find two 
pieces of similar equipment to compare. For example, if the goal is to pick an 
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IETM class for a new feed pump, the user might choose a feed pump with a Class 
II IETM and a feed pump with a Class IV IETM and compare their unaltered 
historical data to see which alternative is better. 

What if one program is larger than the other, however? The user would have to 
convert the total costs of each program to a cost per unit program size to compare 
them directly. But what is a “unit program size”? Three different types of units— 
graduates, equipment units, and IETM units—contribute to the size of a program, 
but not in an easily combinable way. Therefore, the application computes net 
savings in the following manner: 

♦ It computes the unit cost of maintenance for the baseline and the alterna¬ 
tive, then scales their difference by the number of the alternative’s equip¬ 
ment units to get the total net maintenance savings. 

♦ It scales the difference of the costs per graduate by the number of the al¬ 
ternative’s graduates to get the net training savings. 

♦ It scales the difference of the costs per IETM by the quantity of the alter¬ 
native’s IETMs to get the net IETM upkeep savings. 

The sum of these three values, combined with the difference in certain other costs 
between programs, is the total net savings. Thus, to the maximum extent possible, 
the net savings figure reflects the amount of money saved by choosing the alter¬ 
native over a baseline program of similar size. 

For the tutorial example, we chose to assign all three programs the same number 
of graduates per year, the same IETM quantity, and the same number of opera¬ 
tional units. This special case allows the user to compare the baseline and all the 
alternatives directly. If the alternatives have different sizes, they cannot be com¬ 
pared directly; they can be compared only against the baseline. To make a direct 
comparison between two alternatives, the user must run a new analysis with one 
of the two alternatives recast as the baseline. 

Sensitivity Analysis 

You will need to complete the first tutorial to follow along with this tutorial. 

1. Verify that the scenario you used in the Standard Analysis tutorial is 
loaded into the Start sheet: The baseline program should be Program 2, the 
first alternative should be Program 1, the second alternative should be 
Program 3, and the fiscal year should be 1997. 

2. Go to the Variants sheet. 

The Variants sheet allows you to select a high and low variant for each parameter 
loaded onto the Start page. The range is expressed as a percentage of the 
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parameter’s nominal value. For example, if you check the Alternative I table in 
the Start sheet, you would find that the average student pay is essentially constant, 
at $12.63 per hour, throughout the data set (1997 dollars). We think that the pay 
for our program may range between $11.00 and $15.00 an hour. $11.00 is 87 per¬ 
cent of $12.63; $15.00 is 119 percent of $12.63. 

3. Enter 87 percent in the cell for Alternative I with the column heading 
“Average Student Hourly Pay” and the row heading “Low (% Nominal).” 
Enter 119 percent in the same column, in the row with the heading “High 
(% Nominal)”. The Variants sheet should look as it does in Figure 3-17. 

Figure 3-17. Variants Sheet With Student Pay Variants Added 
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4. Make the following additions to Alternative I as well: 


a. Enter 90 percent/110 percent as the low and high variants for 
“Average Instructor Hourly Pay.” 

b. Enter 80 percent/120 percent under “IETM Orientation Training.” 

c. Enter 80 percent/120 percent under “Period Maintenance/Revisions 
Cost.” 

5. Make the following additions to Alternative II: 

a. Enter 80 percent/110 percent under “Average Instructor Hourly Pay.” 

b. Enter 80 percent/120 percent under “Procurement Cost.” 

c. Enter 90 percent/130 percent under “Course Length.” 

6. Go back to the Start sheet, and click the “Run” button under the 
“Sensitivity Analysis” heading. It could take a few minutes to complete 

the run. 

The sensitivity analysis routine works by re-running the standard analysis routine 
for each high and low variant in the Variants page. After the sensitivity analysis 
concludes, the tool also performs an additional standard analysis. The reason for 
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this procedure is that the sensitivity analysis “hijacks” the standard analysis report 
pages to generate its own data. Running the standard analysis after the sensitivity 
analysis ensures that the standard analysis reports and graphs correspond to the 
nominal data on the start sheet. If you run a standard analysis with one set of al¬ 
ternatives and then run a sensitivity analysis with a second set of alternatives, the 
standard analysis graphs and reports will reflect the second set of alternatives. 

7. Click the “Graphs” button in the Start sheet under the “Sensitivity Analy¬ 
sis” heading. 

8. Review the net savings charts for Alternatives I and II. They should look 
like Figures 3-18 and 3-19. 

Figure 3-18. Net Savings Tornado Diagram for Alternative I 
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Figure 3-19. Net Savings Tornado Diagram for Alternative II 


FY: 1997 Alternative 2 Net Savings Tornado Diagram 



Clearly, student pay dominates the outcome. Keep in mind that the data that gen¬ 
erated these graphs are notional; in a real-world case, student pay may have a 
smaller effect, and other variables may have a much larger effect. Given the re¬ 
sults here, however, the analyst obviously would want to research student pay 
further. A smaller range of estimates would greatly improve the certainty of the 
analysis. 

Conversion Cost Utility Worksheet 
(and Variational Study, Part I) 

The conversion cost utility worksheet is a tool that provides a common sense, 
rule-of-thumb method for estimating the cost of converting from a lower class of 
technical manual to a higher-class manual. Although these estimates rely on rules 
of thumb provided by personnel with experience with such conversion projects, 
the inherent variability in these projects and the relatively small amount of data 
about them suggest that actual projects may depart considerably from these rules. 
Because the conversion cost utility builds on these rules (and therefore introduces 
even further variability), it falls under the category of “quick and dirty” tools. 
Nevertheless, it provides a solid theoretical framework for solving a problem that 
has yet to be addressed: converting a Class H or m ETM into a Class IV or V 
IETM. 

This tutorial explains how the utility computes a conversion cost as the user steps 
through the process of creating a conversion cost estimate. It also builds on les¬ 
sons from preceding tutorials to demonstrate how to perform a variational study 
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of one program. The tutorials on the standard and sensitivity analyses compare 
actual programs to another program that serves as a baseline. In this tutorial and 
the one that follows, you will modify the baseline from these tutorials and store a 
modified program. Then you compare it with the original baseline. 

l.a.Open the IETM CBA tool; use the tabs to go to the conversion cost utility 
worksheet (the “ConversionCostWS” tab). 

At the top of this worksheet, you should see “Step 1: Page Estimate.” Although 
the tool can compute estimates for ETM-to-ETM conversions in addition to paper 
manual-to-ETM conversions, it still uses a book paradigm to estimate the size of 
the conversion. The tool uses this approach because all current rules of thumb 
base their estimates on a cost-per-paper-manual-page basis. If a page count is not 
available or does not apply to the manual to be converted, the utility estimates a 
page count on the basis of the size of the manual in kilobytes (K). 

l.b.In the Step 1 section of the worksheet, click on the “Number of paper 
manual pages” in the “Paper or electronic?” radio button list. Then enter 
“20,000” in the cell next to the list. You should see this number appear in 
the cell (Figure 3-20). 

Figure 3-20. Step 1—Conversion Cost Utility 
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1 .c.Notice that there are three boxes in Figure 3-20. The one on the left holds 
the user input. The one on the right holds the final page estimate. The one 
in the middle, with the default button underneath it, holds the kilobyte-to- 
page conversion factor, if needed. To see how this conversion works, se¬ 
lect the “Size in kilobytes” option from the radio button list, and change 
20,000 to 30,000 in the input box. The changes should appear as in Figure 
3-21. 
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Figure 3-21. Step 1—Conversion from Kilobytes to Pages 
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Notice that the units of the input box have changed to K, for kilobytes, in the in¬ 
put box, and k/pg in the conversion box. In addition, the 10 in the default box is 
no longer lined out, meaning that the sheet divides the input value by 10 to get its 
page estimate. 

1. d.The lOkg/pg value is a default. We arrived at this default by looking at 

several ETMs in portable data format (PDF) and comparing their page 
counts with their memory sizes. This number may vary, however, and you 
may feel that another value is more accurate, based on personal experi¬ 
ence. Change the number in the conversion box to 15. Then click the “De¬ 
fault” button. The total pages should reduce to 2,000 when you make the 
change, and then go back to 3,000 when you restore the default value. 

2. Select Class II for the original technical manual. This selection will start 
the conversion project with an electronically indexed ETM of electronic 
page images. The “Initial Complexity” cell should now read $12.50/pg. 

3. a.Select Class IV for the converted tech manual. The “Final Complexity” 

cell should now read $200.00/pg. 

Our “complexity scores” are really just the rules of thumb themselves: the cost- 
per-page of converting from a paper manual to a manual of the designated class. It 
should be possible to enhance these rules of thumb by assigning a relative com¬ 
plexity to the ETM/IETM and converting that complexity to a corresponding cost- 
per-page, and we have designed this utility to handle that eventuality. We have 
seen promising methods that take this approach—most notably an ETM function¬ 
ality matrix for assessing relative complexity. 1 These methods have not yet devel¬ 
oped, however, to the point that we can use them in the utility to estimate costs 
per page. 

3.b.We have seen wide variation in the reported cost per page for converting 
to ETMs of the various classes. For example, we have seen numbers as 
low as $25.00 per page and as high as $250.00 per page for converting 
from paper to a Class IV lE l M. With this in mind, we enable users to 
adjust the values by hand. Set the “Final Complexity” cell in Step 3 to 
$175.00/pg. 

1 See <home.mmcable.com/mandmp/albaiacba/index.htm>. 
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4. Set the rework factor in Step 4 to “Moderate rework expected.” 

The concept of rework stems from the idea that proceeding directly from a lower- 
class ETM to a higher-class ETM is not always feasible. In the tutorial example, if 
we could proceed directly from the original Class II ETM to the final Class IV 
IETM, the expected cost per page is simply $175.00-$ 12.00, or $163.00. That is 
the best case scenario. In the worst case, the team will not be able to save any 
work by using the Class II ETM and will have to start from scratch. The expected 
cost would then be $175.00. Realistically, the cost is likely to fall somewhere 
between these two extremes. The expected amount of rework—converting the 
original ETM’s file format into something compatible with the Class IV’s soft¬ 
ware and so forth—decides where the actual cost falls inside the range provided 
by the two extremes. The .3 value for “Moderate work expected” means that the 
cost will be slightly closer to the best-case scenario than to the worst-case sce¬ 
nario. 

5. The worksheet automatically estimates the conversion cost, based on your 
inputs. It should be slightly less than $500,000 if you have entered every¬ 
thing properly. 

6. a.We estimate that the conversion project will last for three quarters, starting 

in January 1997. Why this date? It is the same start date as that of sample 
program 2 (the Emergency Fill Pump) on the programs sheet. We use the 
results of this tutorial to create a modified program that is based on the 
Emergency Fill Pump program. Enter “1/1997” into the “Start Date” cell 
of Step 6. Enter 3 in the “# quarters” cell. The worksheet should now look 
like Figure 3-22. 


Figure 3-22. Steps 4, 5, and 6 
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6.b.Click the “Compute” button. The tool will spread the total cost of the con¬ 
version into three equal parts over three quarters and adjust them for infla¬ 
tion (Figure 3-23). The “Constant Dollars” column holds the conversion 
cost as expressed in year 2000 dollars. The “Then-Year Dollars” column 
takes the constant dollars and further adjusts them to account for inflation 
during each quarter. The final cost stream for this project should appear as 
shown in Figure 3-23. 

Figure 3-23. Converting Estimate to Then-Year Dollars 
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6.c.If you don’t like a uniform expenditure rate, you can adjust the dollar 
amounts by hand in the “Constant Dollars” column. The Then-Year 
Dollars” column will automatically update to reflect your changes. The 
user is responsible for ensuring that the dollar amounts in the “Constant 
Dollars” column sum to the total conversion cost as given in Step 5. If 
you don’t want to use quarterly time intervals, you can adjust the dates 
manually. 

6.d.Save your work. In the next tutorial, you will use it to create a modified 
program that is based on the second program listed in the Programs sheet. 

Simulation Analysis 

(and Variational Study, Part II) 

In the preceding tutorial, the user estimated a cost for converting from a Class II 
ETM to a Class IVIETM. In the first part of this tutorial, we use that data and 
make a few other assumptions to create a new program, based on a program that 
is used as a baseline in the Standard Analysis and Sensitivity Analysis tutorials. 
The second part of this tutorial uses the baseline and modified programs in a 
simulation analysis. 

1. Open the tool and go to the Programs sheet. Copy all 12 cost reports of the 
program with an Entry ID of 2. Paste them into the first available row of 
data. Make sure there are no blank rows between the existing data and the 
data that you just pasted (see Figure 3-24). 
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Figure 3-24. After Copying Second Program to Programs Sheet 
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2. Every program needs its own unique Entry ID. Because you are creating 
the program directly (as opposed to importing it through the data import¬ 
ing utility) you need to assign this ID by hand. In the Entry ID column, 
change 2 to 4 in every row of the new data. 

Also, in every row of the new data, make the following changes: 

3. In the “Program” column, change the value from ‘TUTORIAL Fill Sys¬ 
tem” to “Tutorial.” 

4. In the “IETM Serial Number” column, change the value to “new.” 

5. In the “IETM Class” column, change the value to 4. The new program 
should now look as it does in Figure 3-25. 
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Figure 3-25. New Program After Tutorial Step 5 
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6. In the “Dev. Cost” column, replace the current values in each of the first 
three rows of the new program with values from the cost estimate gener¬ 
ated in the previous tutorial: In the January 1997 row, enter the value 
161.86; in the April 1997 row, enter the value 162.57; and in the July 1997 
row, enter the value 162.85. You must divide the values provided by the 
conversion cost utility by 1,000 to give them the proper units for the Pro¬ 


grams sheet. 


7. Because both the original and the new program are software-based, as¬ 
sume that the “Period Maintenance/Revisions Cost” column should remain 
unchanged. 


8. Assume a nominal cost to train personnel on how to use the new Class IV 
IETM: In the “IETM Orientation Training” column, enter $10,000 in each 
of the rows for October 1997, January 1998, and April 1997. 


9. For those same three rows, assume a $10,000 cost for converting the C- 
school curriculum and classrooms from the old ETM to the new Class IV 
IETM. (Note that the cost is expressed in $1,000 units, so enter 10.0, not 
10 , 000 .) 


10. Assume no changes in training overhead cost, student pay, instructor pay, 
and student-instructor multiple. The changes in steps 6-9 should make af¬ 
fected columns appear as shown in Figure 3-26. 
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Figure 3-26. Modifications from Tutorial Steps 6-10 
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11. Assume that the passing rate and the number of graduates remain the 
same. Assume that exploiting the enhanced capabilities of the new manual 
allows the school to condense the curriculum by two weeks: In every row 
of the new program, change the value in the “Course Length” column 
from 13 to 11. 


12. In the upper-left comer of the Programs sheet, click the “Update Data” 
button. You have just added a new program to the tool. Click the “Return 
to Start” button. 

13. Now set up the start sheet to compare the baseline program (Program 2) 
with the new program (Program 4). In the Baseline pull-down box, select 
2. In the Alternative I pull-down box, select 4. In the pull-down boxes for 
Alternatives II-IV, select 0 (this selection will make those alternatives 

empty). In the fiscal year (“Use_dollars”) pull-down box, select 

2000. Click the “Load” button. 


The following steps set up and run a simulation analysis that compares the base¬ 
line program (Program 2) to the Alternative I program (Program 4): 

14. Go to the Variants page. Our Alternative I development costs are uncertain 
because we used an estimation tool to compute them. Assume that this es¬ 
timate might be off by ±30 percent: Enter 70 percent/130 percent as the 
low/high variants for the development costs. 
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15. Recall that we chose to keep the costs associated with periodic mainte¬ 
nance and revisions unchanged between Programs 2 and 4. Assume that 
we believe that we can save up to 10 percent of these costs with the new 
manual because its software is easier to work with: Enter 90 percent/100 
percent as the low/high variants for periodic maintenance and revisions 
costs. 

16. Suppose that the figures for IETM Orientation training are very uncertain. 
Perhaps personnel won’t need any additional training on how to use the 
IETM; perhaps costs will be twice as much as originally anticipated: Enter 
0 percent/200 percent as the low/high variants for IETM orientation train¬ 
ing costs. 

17. Suppose there is a possibility that the new IETM will allow the training 
school to enlarge the average classroom size by 15 percent without harm¬ 
ing the quality of instruction: Enter 100 percent/115 percent as the 
low/high variants for the student/instructor multiple. 

18. Finally, LMI’s research has found that some sources believe a Class IV 
IETM will help reduce maintenance costs by improving troubleshooting. 
Others feel that, especially among experienced technicians, maintenance 
costs may go up slightly because the layout of Class IV IETMs makes it 
difficult for experienced users to skip directly to the information that they 
need. Therefore, enter 95 percent/105 percent as high/low variants for the 
maintenance cost per system variable. 

After steps 14-18, the Variants worksheet should look as it does in Figure 3-27. 


Figure 3-27. Variants Sheet Following Tutorial Steps 14-18 
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19. At the upper left comer of the Variants sheet, click the “Return to Start” 
button. 

20. Click the “Run” button under the “Simulation Analysis” heading (but read 
the note below first). 
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Note: The Sensitivity Analysis isolates the effect of changing a variant by chang¬ 
ing only one variable per run. The Simulation Analysis runs every combination of 
variations possible, and can therefore take considerably longer to complete. This 
example uses 243 runs and takes 12 minutes of run-time on a Compaq 266 MHz 
Pentium II laptop computer. 

21. Click the “Graphs” button under “Simulation Analysis” in the Start sheet. 

Select the Net Savings Graph. It should look like Figure 3-28. 

Figure 3-28. Net Savings Results of Simulation 


FY: 2000 


Alternative 1: 
Net Savings 


Simulation Results 



Range 

The graph shows a roughly Gaussian distribution. The mean net savings appears 
to be between $170,000 and $230,000. The median appears to be between 
$230,000 and $260,000. The simulation shows the potential for much larger net 
savings; it also shows a small but not insignificant probability of negative net 
savings. 

22. The simulation analysis, like the sensitivity analysis, reruns the standard 
analysis after it is complete. This procedure allows the user to evaluate the 
nominal net savings. Return to the start page, click on the Standard Analy¬ 
sis’s “Chart” button, and choose the net savings chart at the bottom of the 
menu (Figure 3-29). 

This chart shows nominal net savings of approximately $175,000 after the first 12 
quarters. A review of the other graphs from the standard analysis menu shows that 
the net savings is very negative in the first three quarters because of the develop¬ 
ment costs. Eventually, it becomes positive because the two weeks of reduced 
course length create significant savings over the baseline. 
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Figure 3-29. Net Savings of Nominal Case 


FY:2000 


Net Savings From Baseline 



Program Statistics Utility 

The Program Statistics utility computes, for a given program or subset of pro¬ 
grams, the mean and variance of a user-requested variable. The utility will not be 
very useful unless sufficient data exist in the Programs sheet to provide a mean¬ 
ingful amount of variation across IETM classes and categories. Therefore, this 
tutorial is limited to showing the user how to activate the utility and make a query. 
The tutorial discusses the types of queries that the tutorial can make and then 
steps you through a single-program example. 

1. Go to the Programs sheet. 

2. Click the “Generate Statistics” button (the third button at the top left cor¬ 
ner of the Programs sheet). You should see the Programs Statistics form 
appear, as in Figure 3-30. 

3. Click the “Program Category” pull-down menu; the choices read “All 
Programs,” “Hull-Mechanical-Electrical” “InformationTechnology,” and 
“Weapons Systems.” Click the “Class” pull-down menu; the choices read 
“All Classes,” “0,” “1,” “2,” “3,” “4,” “5.” 
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Figure 3-30. Program Statistics Utility 
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The utility allows users to calculate statistics on a subset of programs, based on 
the program category and IETM class. Selecting “All Programs” in the “Program 
Category” pull-down menu and “All Classes” in the “Class” pull-down menu will 
ensure that all programs are selected. Selecting “All Programs” and “4” will en¬ 
sure that all Class IV programs are selected. Selecting “Information Technology” 
and “3” will ensure that all Class ID programs of Information Technology systems 
are selected. More advanced queries—such as “Select all Information Technology 
systems at or above Class ID”—are not possible. By the time sufficient data exist 
to make such queries useful, the tool should be enhanced so that the program data 
come directly from a database. The database would provide the capability to per¬ 
form advanced queries for statistical or other calculations. 

4. Select “Training OH Cost” from the “Variables” pull-down menu. Leave 
the default value (2,000) in the “Fiscal Year” pull-down menu. 

5. Click the “Single Program” button. When you are prompted for a unique 
program ID, enter 1. 

Selecting “Single Program” instructs the utility to compute the mean and variance 
of the specified variable for one program only; in this case, the statistics provided 
show the mean and variance of the 12 quarters of training overhead costs for the 
first program on the programs sheet. In single program mode, the utility ignores 
the “Program Category” and “Class” program values. 

6. Review the results, which should appear as in Figure 3-31. 
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Figure 3-31 . Calculation Results 
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Appendix 

Description of Variables 


This appendix describes each of the 23 variables that are characteristic of a pro¬ 
gram cost report. A cost report is a row of data in the Programs sheet. Each vari¬ 
able makes up a column of the Programs sheet. When the variable name differs 
from its column heading as it appears in the Program sheet, we list the column 
heading in parentheses after the variable name. 

♦ Entry ID. This is a unique number, similar to a database key, that uniquely 
identifies a set of cost reports as belonging to a single program. Every cost 
report in a program should have the same entry ID. Each entry ID must 
correspond to exactly one program. 

♦ Program Name. The overall system for which the technical manual has 
been created. In other words, a manual that describes the avionics package 
on an F-18 might have “F-18” as its program name. A manual that de¬ 
scribes the F-18’s landing gear would have the same program name. Each 
cost report must include a program name. 

♦ Subsystem Name. The specific piece of equipment that the technical man¬ 
ual describes. For example, the F-18 avionics system would use the name 
of the avionics system as the subsystem name. Each cost report must in¬ 
clude a subsystem name. 

♦ IETM Serial Number. This is a unique number or string that uniquely 
identifies the IETM (or ETM or paper manual). The tool does not use this 
variable to identify a technical manual; it is included to provide users with 
an opportunity to identify the manual with a familiar number. 

♦ Program Category (Category). The classification of system to which the 
technical manual applies: Hull/Mechanical/Electrical, Information Tech¬ 
nology, or Weapons Systems. The tool does not use this variable directly 
in the analysis but may use it to gather statistics on a subset of the data. 

♦ IETM Class (actually, applies to any technical manual). The class of the 
manual as given in Jorgensen (1994). This value is an integer from 0 to 5: 
0 corresponds to a paper manual; 1-5 corresponds to a Class 1-5 elec¬ 
tronic technical manual. The tool does not use this variable directly in the 
analysis but may use it to gather statistics on a subset of the data. 

♦ Report Date. The date of an individual cost report. The time interval be¬ 
tween consecutive reports must be consistent within each program and 
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across all of the programs used in an analysis. See Chapter 2 for a more 
detailed explanation of cost report time intervals. 

♦ Procurement Cost. The cost associated with purchasing the manual from a 
developer. These costs should be fees that are not specified elsewhere in 
this sheet. 

♦ Development Cost (Dev. Cost). The cost associated with developing the 
manual—any original authoring, plus costs related to converting an ETM 
from a paper manual or lower class of ETM. In some cases, the user may 
consider the development cost to be a part of the procurement cost; in this 
case, the user should not list the development cost separately. (The pro¬ 
curement cost should include the total value, and the development cost 
should be zero.) 

♦ Periodic Maintenance and Revisions Costs (Period Mainte¬ 
nance/Revisions Cost). Costs related to periodically revising the manual to 
reflect changes in content, as well as any patches/fixes to ETM software. 

♦ C-School Conversion. Costs associated with upgrading a C-school to ac¬ 
commodate the use of the technical manual. These costs are restricted to 
funds required to bring a new manual into the classroom and display it in 
that setting. It does not include the general costs of converting the entire 
classroom environment to an electronic, multimedia format. If the 
C-school curriculum and classroom format are not modified to accommo¬ 
date using the manual, these costs should be set to zero. 

♦ Training Overhead Cost (Training OH Cost). Overhead costs associated 
with running a C-school that uses the technical manual. This cost is diffi¬ 
cult to define and therefore is not used in the analysis at this time. (The 
variable is a placeholder.) If the user is unsure of these costs, they should 
be set to zero. 

♦ Average Student Hourly Pay (Avg. Student Hourly Pay). Average total 
hourly compensation of a student undergoing training on the subsystem 
that the manual describes. This figure includes per diem, housing allow¬ 
ance, and other allowances. 

♦ Average Instructor Hourly Pay (Avg. Instructor Hourly Pay). Average to¬ 
tal hourly compensation of an instructor on the subsystem that the manual 
describes. This figure includes per diem, housing allowance, and other al¬ 
lowances. 

♦ Student/Instructor Multiple. Number of students per instructor (i.e., aver¬ 
age class size) in the training course for the subsystem that the manual de¬ 
scribes. This number must be greater than zero. 
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♦ Pass Rate. The ratio of graduates to enrolled students in the current class 
of the course. This number must be a percentage greater than zero and less 
than or equal to 100. 

♦ Course Length. The length of the training course for the subsystem that 
the manual describes. The course length should be the total time from the 
start of the course to the end, not the number of weeks in the current time 
interval when classes were in session. 

♦ Number of Graduates (Grads). The total number of graduates in the cur¬ 
rent time interval from the training course for the subsystem that the man¬ 
ual describes. This number must be a positive integer. 

♦ Maintenance Cost Per System or Unit (Maint. Cost/System). The average 
cost of maintenance on all units of the equipment that the manual de¬ 
scribes. (For example, if 100 pumps required a total of $50,000 for main¬ 
tenance in the current time interval, this value should be $500.) 

♦ Systems or Units in Service (Systems in Service). The total number of 
units in service during the time interval. This number, multiplied by the 
Maintenance Cost Per System or Unit, should give the total dollars spent 
on maintaining these units in the given time interval. This value includes 
units that are in a reduced status or out-of-commission. This number must 
be a positive integer. 

♦ IETMS in Service. The total number of manuals in service for the appro¬ 
priate subsystem during the current time interval. This figure applies to 
paper manuals and noninteractive ETMs as well. This number must be a 
positive integer. 

♦ Readiness. The percent readiness of the units in service for this time inter¬ 
val. This variable is not used in the analysis; it is a placeholder. However, 
you must enter a number between zero and 100. 
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